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(54) Information processing apparatus, method, and computer-readable medium 



(57) A system and method for creating multimedia 
information synchronized with a video signal (a p., a pro- 
gram of a television broadcast) are provided. The mul- 
timedia information may be formatted in accordance 
with, for example, the MHEG standard or the ATVEF 
standard. The system includes means for specifying 
and verifying the contents of, and synchronizations for. 
the multimedia information. A scenario creation pro- 



gram creates a scenario composed of a plurality of 
scenes synchronized with playt>ack of the video. A 
scene creation program combines objects consisting of 
text, images, audio, operation buttons and the like to 
create scenes (pages). A contents management pro- 
gram, a scenario management program and a scene 
management program set a directory of the contents, 
the scenarios and the scenes. 
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Description 

[0001] The present invention relates to an information 
processing apparatus, method and computer-readable 
medium. 5 
[0002] Standards exist, or have been proposed, for 
displaying infornnation and data, including moving imag- 
es, still images, sound ard text in response to a user's 
interactive commands, and for synchronously display- 
ing such information and data. One such standard, pro- io 
posed by the International Organization for Standardi- 
zation (ISO), is that defined by the Multimedia and Hy- 
permedia Infonnation Coding Experts Group (MHEG). 
Another standard is that defined by the Advanced Tele- 
vision Enhancement Forum (ATVEF), a consortium of '5 
broadcast and cable networks, consumer electronics 
companies, television transport operators and technol- 
ogy companies. 

[0003] Information such as that corresponding to the 
MHEG standard or the ATVEF standard (hereinafter re- 
ferred to generally as "multimedia information*) can be 
provided to a viewer in a television broadcast. Other 
methods of transmission and display are contemplated, 
however, including over the Internet to computers. 
[0004] Multimedia information generally changes little 2S 
with respect to the video to which it correspor>ds. To pro- 
vide multimedia information at an appropriate time, how- 
ever, the multimedia information should be synchro- 
nized with the video to which it corresponds (for exam- 
ple, the moving images of a television broadcast). An 30 
authoring system or authoring tool for creating such syn- 
chronized multimedia information, however, does not 
exist. An object of the present invention is to provide 
such an authoring system or authoring tool. 
[0005] Various respective aspects and features of the 
invention are defined in the appended claims. 
[0006] Embodiments of the invention can provide an 
information processing system and method. The system 
includes means for specifying multimedia information 
and video with which the multimedia information is syn- 4o 
chronized, means for creating the multimedia informa- 
tion and means for setting the timing for displaying 
scenes of the multimedia information to synchronize this 
information with the video. The method includes speci- 
fying multimedia information and video with which the **5 
multimedia information is to be synchronized, creating 
the multimedia information and setting timing for dis- 
playing scenes of the multimedia information to syn- 
chronize this information with the video. A computer- 
readable medium containing code for executing this so 
method also is provided. In preferred embodiments, the 
multimedia information is either MHEG information or 
ATVEF information. 

[0007] The specifying means preferably comprises a 
contents creating section for identifying the contents of 5S 
the multimedia information and of the video with which 
the multimedia information is to be synchronized. The 
creating means preferably comprises a scene creating 



section for creating scenes of the multimedia informa- 
tion, and the setting means preferably comprises means 
for identifying the video's starting time for displaying 
scenes of the multimedia information synchronized with 
such starting time. 

[0006] The information processing system preterabty 
also comprises a display, and a display controlter. for 
displaying a time axis of the video and symbols corre- 
sponding to a hierarchy of the scenes. The information 
processing system preferably also comprises means for 
setting a synchronization flag for activating or deactivat- 
ing synchronized playback of the multimedia informa- 
tion with the video. A mid-processing playback flag for 
allowing or disallowing mid-processing display of an im- 
age also preferably is provided. Disallowing such mkf- 
processing display prevents the image's display to a 
viewer who begins viewing the video at a point when the 
image's display would be meaningless. 
[0009] The information processing system preferably 
also comprises managing means for organizing, in a hi- 
erarchical manner, the multimedia information into con- 
tents, scenark>s and Irenes and for managing this hier- 
archkral structure. Means for setting the priority level of 
a scene, such that the rate at which the scene is trans- 
mitted from a broadcast station corresponds to a view- 
er's likelihood of selecting the scene for display, also 
preferably is provided. 

[0010] The infornnation processing apparatus prefer- 
ably also comprises means for setting a display mode 
flag for determining whether a viewer can control the 
multimedia information's display. The setting of this flag 
prevents the viewer from switching off this display. 
[0011] The invention will now be described by way of 
example with reference to the accompanying drawings, 
throughout which like parts are referred to by like refer- 
ences, and in which: 

Fig. 1 IS a block diagram of a system for broadcast- 
ing multimedia information created using the appa- 
ratus and method of an embodiment of the present 
invention; 

Fig. 2 is a bkKk diagram of an apparatus for creating 
multimedia information. 

Fig. 3 is a block diagram of the sections of a com- 
puter program executed by the multimedia creating 
apparatus of Fig, 2. 

Fig. 4 is an illustratbn of the objects of a scene. 
Fig. 5 is an illustration of a directory for the multi- 
media informatk^n showing the information's hierar- 
chical structure. 

Fig. 6 is a block diagram of IRD 4 of Fig. 1 . 
Fig. 7 is a bkx;k diagram of the sections of a com- 
puter program executed by IRD 4 for playing back 
multimedia informatbn. 

Fig. 8 is a flowchart III ustrating the steps for confirm- 
ing the synchronizatbn settings created by the mul- 
timedia creating apparatus of Fig. 2. 
Fig. 9 is an example of a display appearing on dis- 
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play apparatus 1 5 of the muttimedia creating appa- 
ratus of Fig. 2 pursuant to steps SI and S3 of Fig. 6. 
Fig. 10 is a flowchart illustrating the steps tor con- 
firming the interactive setting process created by 
the multimedia creating apparatus of Fig. 2 
Fig. 11 is an illustration of the link structure of 
scenes. 

Fig. 1 2 is an example of a display appearing on dis- 
play apparatus 15 in accordance with the process 
of Fig. 10 for confirming interactive settings. 
Fig. 13 is an illustration of the effect of the setting 
of synchronization flags. 

Fig. 14 is an illustration of the effect of the setting 

of rtiijd-processing display flags. 

Fi^: 15 illustrates a sending rate for scenes. 

Fig. 16 further illustrates a sending rate for scenes. 



by CPU n . As indicated above, in this example, rt is 
assumed that the authoring program is for authoring 
MHEG information. Although each of these blocks re- 
ceives information from one or more of the other bkxks 

5 and provides information to one or nx)re of the other 
blocks, for clarity of presentatbn, arrows among the 
bkx:ks indcating such inputs and outputs are omitted. 
[0017] A contents creation section 21 creates con- 
tents information for the MHEG information. This con- 

10 tents information corresponds to a video {e.g., a pro- 
gram of a televisk>n broadcast) to be synchronized with 
the MHEG information. The contents information is pro- 
vkJed to the contents management section 24. 
[0016] The contents infornnation corresponds to a 

IS group of scenarkjs and comprises eight items of infor- 
mation: 



[0012] ; A broadcasting system for broadcasting video (1) contents_name 
and multimedia information, such as, e.g., MHEG infor- (2) sen/ice_name 
mation or ATVEF information, is shown in Fig. 1 . Broad- (3) program_name 
cast station 1 transmits television broadcasting signals (4) es_number 
containing multiplexed multimedia information signals, (5) start_time 
Antenna 3 receives these signals via satellite 2. The re* (6) duration 
ceived signals are supplied to IRD (Integrated Receiver (7) additional Jnfo 
and Decoder) 4 which extracts the television and multi- (S) application_number 
media signals on a predetermined channel. The extract- 
ed signals are supplied to display apparatus 5 for dis- [0019] In the contents_name, 
playing the multimedia information superposed upon ■contents_1988_4_1_1," for example, is recorded to 
the video of a television broadcast. kJentify the name of the contents. In the service_name. 
[0013] Fig. 2 is a block diagram of an apparatus in 30 "NHK,' for example, is recorded to identify the channel 
accordance with an embodiment of the present inven- on which the program including the multimedia informa- 
tion for creating multimedia infornrration for transmission tion is broadcast. In the program_name, "baseball.' for 
by broadcast station 1 . The apparatus of Fig. 2 can be example, is recorded to klentify the program's name. In 
used to create any form of multimedia information, for the es_number, "lO." for example, is recorded to identify 
example; MHEG information or ATVEF information. For 35 the number of elementary streams. In the startjime, 
the purpose of this example, however, it will be assumed "+00/0O/O0.* for example, is recorded to kSentity the rel- 
that the apparatus of Fig. 2 is used to create MHEG in- ativo time for starting the display of the MHEG infonna- 
formation. tion with respect to the video with which the MHEG in- 
[0014] The apparatus of Fig. 2 includes CPU 11 for formation is synchronized. In the duration, 'OO/OO/OO,' 
providing overall control in accordance with a program 40 for example, is recorded to indicate the duration of the 
stored in ROM 12 and RAM 13. CPU 11 may be. e.g., a multimedia informatran's display from the relative start 
microprocessor or similar device. CPU 11 alsoexecutes time. In the additionaljnfo, 'scenark? 1 for example, is 
a muttimedia authoring program stored on, e.g., hard recorded to indicate that the information corresponds to 
disk 14 for creating MHEG information. The authoring the first scenario. In the application_number, "S." for ex- 
program can. of course, be stored on any computer- ^5 ample, is recorded to indicate the number of scenarbs 
readable medium accessible to CPU 11 , e.g.. a floppy contained in the contents 

disk, magnetic tape, optical disk. etc. [0020] The contents creation section 21 also manag- 

[0015] Display apparatus 15 displays the MHEG in- es an applicatbn information file for recording the start 

formation, and the video to be synchronized with the time and duration for broadcasting a scene (described 
MHEG information, to an operator. Keyboard 16 and so later with reference to Fig. 9). 

mouse 17 enable the operator to control the apparatus' [0021] The scenarc creation section 22 creates a 
operations. A communication section 18 including, for scenario, and scenario informatran. and provides its out- 
example, a modem, controls communcations over ate!- put to the scenario management section 25. Ascenark) 
ephone line, the Internet, or the tike. Sound playback is composed of a plurality of scenes synchronized with 
apparatus 19 ptays back sound to be synchronized with ss the video. The scenario informatbn comprises four 
the video or MHEG information. items of information: 
[0016] Fig. 3 shows the functional blocks (sections) of 
the multimedia information authoring program executed ( 1 ) GS_name 



3 



5 



EP 0 942 595 A2 



6 



(2) data_type 

(3) customizedjnfo 

(4) scene_number 

[0022] In the es_number. 'DATA 1,' for example, is 
recorded to identify the name of the elementary stream 
corresponding to the scenario. In the data_type, "mheg, 
■ for example, is recorded to identify the type of multi- 
media information. In the customized_info, any specific 
information corresponding to the scenario may be re- 
corded. In the scene_number, "3," for example, is re- 
corded to indicate the number of scenes in the scenario. 
[0023] The scene creation section 23 combines "ob- 
jects* As shown in Fig. 4, the objects make up a scene 
(page) and may be textual information, image informa- 
tion, audio informal ion, operational buttons, and the like. 
The output of scene creation section 23 is provided to 
scene management section 26. 
[0024] A scene can be switched on and off either by 
synchronization signals contained in the video signal or 
through a viewer's interactive operations, for example, 
activating operational buttons displayed in the scene. 
Further, it is possible to set, in the operational buttons, 
functions for executing predetennined processing, for 
example, to provide textual information to be displayed 
for applying for the purchase of a product. 
[0025] A scene is composed of scene data and six 
items of information corresponding to these data: 

(1) scene_priorityJevel 

(2) fil0_name 

(3) stream_event_name 

(4) stream_event_time 

(5) stream_name 

(6) es_name 

[0026] In the scenc_pnorityjGvel, "1 , ' for example, is 
recorded to identify the sending rate (described later) 
for the scene. In the file_name. "bitmap.jpg,* for exam- 
ple, is recorded to identify the name ot an object. In the 
stream_event_name. 'stream event 1,* for example, is 
recorded to Identify the name of the stream event. In the 
stream_event_time, "00/00/00, ' for example, is record- 
ed to identify the starting time for sending the scene. In 
the slream_name. "slream link 1,' for example, is re- 
corded to identify the name of the stream. In the 
es_name, 'DATA 1 *. for example, is recorded to identify 
the name of the elementary stream of a link destination. 
[0027] The scene creation section 23 also manages 
scene information and records within this information 
synchronization flags (described bek)w with reference 
to Fig. 13). mid-processing playback nrvxle flags (de- 
scribed below with reference to Fig. 14) and display 
mode flags (described below). 
[0028] The contents management section 24, the 
scenario management section 25 and the scene man- 
agement section 26 set the directory for the contents, 
the scenarios and the scenes. An example is shown in 



Fig. 5. In this example, a contents Cl is set in the high- 
est-order dirocioiy, and scenarios SCI, SC2 and SC3 
are set in the next high est -order directory. In the scenar- 
io SC1, three scenes, scene 1, scene 2 and scene 3, 
5 are set. In the scenarios SC2 and SC3, scene 4 and 
scene 5. respectively are set. 

[0029] Referring again to Fig. 3, a time control section 
27 reads time information for the video to be played back 
by video playback section 29 and outputs this informa- 
10 Won to a video and MHEG display control section 31 . An 
interactive control section 28 records settings for switch- 
ing the display of the scenes in response to operations 
of the user. 

[0030] The video playback section 29 plays back the 
IS vkJeo with whch the MHE G information is to be synchro- 
nized. The MHEG playback section 30 plays back the 
MHEG information to verify that the creation, synchro- 
nizations, etc. are correct. The video and MHEG display 
control section 31 controls the display apparatus 15 
20 (Fig. 2) 60 that the MHEG information played back by 
the MHEG playback section 30 is displayed superposed 
over the video information played back by the video 
playback section 29. 

[0031] Fig. 6 shows the construction of IRD 4 (Fig. 1 ) 
^5 tor playing back broadcasted video and multimedia in- 
formation to a viewer A receiving processing section 41 
containing a down converter, a tuner, a demodulator, a 
descrambler and the like, extracts predetennined chan- 
nel components (television broadcasting signals and 
30 MHEG information signals) from the broadcasted sig- 
nals received on antenna 3 and supplies these compo- 
nents to CPU 42. 

[0032] CPU 42 controls IRD 4 through execution of a 
control program, stored in ROM 43 and RAM 44, and a 

35 MHEG information playback program stored on hard 
disk 45. The broadcasted video and MHEG informatkxi 
are displayed on display apparatus 5 (Fig. 1). 
[0033] A display control section 46 controls superpos- 
ing of the MHEG information over the broadcasted video 

40 displayed on display apparatus 5. Keyboard 47. mouse 
48. and remote controller 49 provide input from a viewer 
A communication section 50, tor example, a modem, 
controls information communications over a telephone 
line, the Internet and the like. 

45 [0034] Fig. 7 shows the functional blocks of the MHEG 
information playback program for IRD 4 stored on hard 
disk 45 and executed by CPU 42. A video playback sec- 
lion 54 plays back the video of a channel selected by 
the viewer. A MHEG playback section 55 plays back 

^0 MHEG information corresponding to this video. A video 
display control section 56 controls the display apparatus 
5 (Fig. 1). This control causes the MHEG information 
from MHEG playback section 55 to be superposed over 
the video from vkJeo playback section 54. 

55 [0035] A directory management section 51 manages 
the MHEG information's directory and provides this in- 
formation to the MHEG playback section 55 in accord- 
ance with predetermined liming. A timing control section 
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52 performs time measurement using, as a reterence, 
the time at which the video playback section 54 starts 
playback of the video corresponding lo ihe MHEG infor- 
mation and informs the MHEG playback section 55 of, 
for example, the slarljime contained in the contents in- 5 
fomr^tion. 

[0036] An interactive control section 53 detects oper- 
ations by a viewer upon the keyboard 47, the mouse 48 
or the remote controller 49, and performs predetermined 
processing pursuant to these operations, such as io 
switching a scene. 

[0037] Referring again to the apparatus for creating 
multimedia information shown in Fig. 2, by operating the 
keyboard; 16 (Fig. 2) and setting predetermined values 
in the pontents information (e.g., startjime and dura- ^5 
tion), a scene can be set to be interactively synchronized 
with the Vkleo defined in program_name. These settings 
then c^in.be confirmed by CPU 11 in response to an op- 
erator'^ instructions received from keyboard 16 or 
mouse 17. In response, CPU 11 performs the process- 
ing shown in the flowchart of Fig. 8. 
[0038] As shown in this Figure, in step SI, the time 
control section 27 (Fig. 3) reads the time information 
(start time and end time) of the video to be played back 
by video playback section 29. This time information Is 2S 
provided to the video and MHEG display control section 
31. As shown in Fig. 9. the video and MHEG display 
control section 31 causes display apparatus 15 to dis- 
play a time axis for the vkleo based upon the input time 
infonmatkxi. 30 
[0039] In step S2. the time control section 27 (Fig. 3) 
reads the playback start time (start_time) and the play- 
back duration (duration) for a scenario to be synchro- 
nized with the video. This information also is provided 
to video and MHEG display control section 31 . The op- 
erator also provides the sending start time (time that 
broadcasting begins) and the sending duration (duration 
of broadcasting) tor the MHEG data. This information is 
recorded in the appltcatbn information file of the con- 
tents creation section 21 . The time control section 27 
also reads this information and provides it to the video 
and MHEG display control section 31 . 
[0040] In step S3, the video and MHEG display control 
sectnn 31 causes display apparatus 15 to display the 
playback start time, the playback duration, the sending -'S 
start time and the sending duratk>n along the time axis 
displayed in step S1. This display, shown in Fig. 9. en- 
ables the operator to visualize, and confirm, the syn- 
chronizatran settings for the MHEG informatbn. 
[0041] In the example shown in Fig. 9, the playback so 
start times, the playback durations, the sending start 
times and the sending durations for the scenark^s SCI 
and SC2 are set with respect to the start time lor the 
corresponding video program. For scenario SCI . in this 
example, the playback start time is set for the same time 5S 
as the program start time, the playback duration is set 
for 1 0 minutes, the sending start time is set for 5 minutes 
before the program start time and the sending duration 



is set for 20 minutes. For scenario SC2, the playback 
start time is set for 20 minutes after the program start 
time, the playback duratkxi is set for 10 minutes, the 
sending start time is set for 1 5 minutes after the program 
start lime, and the sending duration is set for 20 minutes. 
[0042] During the 5 minute duratkxis after completkxi 
of playback of the scenarios, a viewer can, for example, 
make a telephone call to purchase an item that was dis- 
played (in the case of a shopping program). Since the 
MHEG creating apparatus enables recording of the set- 
lings such that the MHEG information is transmitted only 
during the necessary times, the transmission channel is 
used effectively for transmitting a plurality of MHEG data 
in a time divisional manner. 

[0043] The process of confirming the interactive set- 
tings for the MHEG information prepared by the creatkxi 
apparatus of Fig. 2 is described with reference to Figs. 
1 0, 1 1 and 1 2. This process enables the user to visual- 
ize, and confirm the accuracy of, the scenes* interactive 
relationships with respect to time (link relationships) af- 
ter the synchronization settings, such as playback start 
time and the like, have been set and the scenes within 
the scenarb have been linked. As an example, as 
shown in Fig. 11, assume that scenes 2 through 5 are 
set as possible selectbns for display by the viewer after 
scene 1 and that scenes 6 through 6 are set as possible 
selections for display by the viewer after scene 2. 
[0044] Upon receipt of an instructk)n for starting the 
confirmatran process on keyboard 16 or mouse 17 (Fig. 
2), the time control section 27 (Fig. 3) reads, in step S21 
(Fig. 10). the time information (start time and end time) 
for the video to be played back by the video playback 
section 29 (Fig. 3) and supplies this information to the 
vkieo and MHEG display control section 31 (Fig. 3). As 
shown in Fig. 12, the video and MHEG display control 
section 31 causes display apparatus 15 (Fig. 2) to dis- 
play a time axis for the video based upon the received 
time information. 

[0045] In step S22, the lime control section 27 next 
reads the playback start lime and the playback duration 
for the first scene (scene 1 ) and supplies this informatbn 
to the video and MHEG display control sectbn 31. In 
step 823, as shown in Fig. 12, the video and MHEG dis- 
play control section 31 causes display apparatus 15 to 
display, below the time axis for the video, a symbol in- 
dicating the playback duration for scene 1 . This symbol 
starts 10 minutes after the video start time. The video 
and MHEG display control section 31 also causes dis- 
play apparatus 15 to display a symbol for scene 2 adja- 
cent to the symbol for scene 1. The video and MHEG 
display control section 31 further causes display appa- 
ratus 15 to display a triangular mark within the symbol 
for scene 2 to indicate thai other scenes are present 
(possible other selections for display after scene 1) in 
the same hierarchy as that for scene 2. 
[0046] In step S24. the interactive control section 28 
determines whether the user has selected the symbol 
for scene 2 (e.g. , clicked by the mouse 1 7). If this sectbn 
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doiormines that the symbol for scene 2 has been select- 
e<i, ihG process proceeds to step S25. 
[0047] In step S25, the video and MHEG display con- 
trol section 31 pulls down the symbols for scenes 3 
through 5, as shown in Fig. 12, and causes display ap- 5 
paratus 15 to display these symbols below the symbol 
for scene 2. 

[0048] In step S26. the interactive control section 28 
determines whether the user has selected one of the 
symbols for scenes 2 through 5. If this section deter- io 
mines that one of these symbols has been selected, the 
process proceeds to step S27. In this example, it is as- 
sumed that the symbol for scene 2 has been selected. 
[0049] In step S27, as shown in Fig, 1 2, the video and 
MHEG display control section 31 causes the display ap- 
paratus 15 to display the symbol for scene 2 below the 
symbol for scene 1 and further display the symbol for 
scene 6, which is linked to scene 2, in a symbol adjacent 
to the symbol for scene 2. Within the symbol for scene 
6. a triangular mark is displayed to indicate that other ^ 
scenes are present (possible other selectbns for display 
after scene 2) in the same hierarchy as that for scene 6. 
[0050] In step S28, the Interactive control section 28 
determines whether the symbol for scene 6 has been 
selected. If this section determines that the symbol for 2S 
scene 6 has been selected, the process proceeds to 
step S29 where the video and MHEG display control 
sectbn 31 pulls down the symbols for scenes 7 and 8 
and displays these symbols below the symbol for scene 

6. 30 

[0051] In step S30. the interactive control section 26 
determines whether one of the symbols for scenes 6 
through 8 has been selected. If this section determines 
that one of these symbols has been selected, the proc- 
ess proceeds to step S31 . In this example, it is assumed ^ 
that the symbol for scene 6 has been selected. 
[0052] In stop S31 , the video and MHEG display con- 
trol sectran 31 displays the symbol for scene 6 below 
that for scene 2. Since no scenes are present in a hier- 
archy below that tor scene 6, the processing terminates, 
[0053] The contents creation section 21 (Fig. 3) also 
enables a user to set synchronization signals (flags) 
within the stream ot video information. These flags are 
set in the scene information file by the scene creation 
section 23 and are processed by I RD 4 (Fig. 1 ). The set- 
ting of a synchronization flag for a scene to the on state 
causes the scene's switching to be synchronized with 
the video. The setting ot such a flag to the off state dis- 
ables such synchronization with the video. 
[0054] For example, as shown in Fig. 1 3, assume that 50 
scenes 1 through 4 form scenario SCI and a synchro- 
nization flag for switching to scene 3 in the middle of 
scene 2 is received from the video signal. Assume fur- 
ther that in pattern 1, the synchronization flag is oft and 
in pattern 2, the synchronization flag is on. As shown in ss 
Fig. 13, in pattern 1 (synchronization flag off), after 
scene 2 is terminated, scene 3 is played back (the play- 
back is not synchronized with the video). In pattern 2 
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(synchronization flag on), although playback of scene 2 
ts incomplete, the scene nevertheless is swrtched to 
scene 3 (synchronized playback is performed). 
[0055] The scene creation section 23 also may set a 
mid-processing playback rrode flag in the scene infor- 
nnation file For example, as shown in Fig. 14. assume 
that scenes 1 through 3 form scenario SCI, that these 
scenes are set to be played back in sequence and that 
a viewer's playback of the video signals begins during 
mid-processing of scene 2. Assume further that, in pat- 
tem 3, the mid-processing playback mode flag is set to 
the on state and in pattern 4, the mid-processing play- 
back mode flag is set to the off state. As shown in Fig. 

1 4, in pattern 3, the playback of scene 2 is started during 
the mid-processing of this scene (since the mid- 
processing playback mode flag is on, a mid-processing 
display occurs). In pattern 4, on the other hand, when 
playback ot the video signals begins (for example, IRD 
4 is turned on and starts to decode received data), the 
play back of scene 2 does not begin even If scene data 
are transmitted (since the mid-processing playback 
mode flag is off, a mid-processing display does not oc- 
cur). The first scene played t>ack is scene 3 whch be- 
gins after terminatcn of the lime for playback of scene 
2 and at the time set for starting play back of scene 3. 
This flag prevents the display of multimedia information 
which is meaningful only if displayed in synchronization 
with the video- 

[0056] The scene creation section 23 also can set a 
display mode flag for transmission to IRD 4 to disable a 
viewer's ability to tum off multimedia information other- 
wise selectable by the viewer. For example, this flag 
may be used it the scene is a commercial synchronized 
with a commercial video and display of the multimedia 
information on display apparatus 5 is desired regardless 
of the viewer's intent. If tho display mode flag is set to 
the on state, the scene is displayed regardless of the 
viewer's selection on the controls for IRD 4. II the display 
mode flag is set to the off stale, the viewer's selection 
on the controls for IRD 4 determines whether the MHEG 
informatran is displayed. 

[0057] The scene creation section 23 also can set, in 
the scene_prority_level tor the scene, a number to de- 
termine the rate at which the scene is transmitted from 
broadcasting system 1 . For example, as shown in Fig. 

15, if scenes 1 through 3 form scenarka SC1 and scene 
1 (the initial scene) is selected for display most frequent- 
ly by viewers, the sending rate for scene 1 can be set 
to, e.g., '2' and the sending rate for scenes 2 and 3 can 
be set to. e.p./r. As shown in Fig. 1 6, in the information 
stream for scenario SCI whose transmission rate is set 
in this manner, the information for scene 1 is sent twice 
as frequently as that for scenes 2 and 3. This transmis- 
sbn rate decreases the time before the most frequently 
viewed scene (scene 1 ) is displayed to the viewer upon 
selection. Although a ratio of integers is illustrated for 
the sending rate, this setting could be any ratio. 
[0058] Although the synchronization settings for the 
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MHEG information with respect to the video are de- 
scribed above as being set using relative time, these 
synchronization settings also could be set with respect 
to real or absolute time. 

[0059] A computer program lor performing the above- s 
described nnethods may be stored on any computer- 
readable medium, such as an optical disk, magnetic 
tape or CD-ROM. and such a program may be distrib- 
uted through any means of distribution such as the In- 
temet or digital satellites. io 
[0050] In summary, embodiments of the invention can 
provide an information processing apparatus, method 
and computer-readable medium for creating, displaying 
and synchronizing audio, pictorial and textual data with 
video infonmation. is 



Claime 

1. An informatbn processing apparatus for creating 20 
multimedia information to accompany video infor- 
mation transmitted tor display to a viewer, said in- 
formation processing apparatus comprising: 

creating means for creating said multimedia in- 25 
formation; 

identifying means for identifying video informa- 
tion with which said multimedia information is 
to be synchronized; and 

setting means for setting the timing for display- 30 
ing one or more scenes of said multimedia in- 
formation so as to cause said displaying to be 
synchronized with the display of said video in- 
formation. 

3S 

2. An intormation processing apparatus as in claim 1, 
wherein said multimedia information conforms to 
the MHEG standard. 

3. An information processing apparatus as in claim 1, 40 
wherein said multimedia information conforms to 

the ATVEF standard. 

4. An information processing apparatus as in claim 1, 
wherein said setting means comprises means lor 
setting said timing with respect to the relative time 
for the display of said video information. 

5. An information processing apparatus as in claim 1 . 
further comprising: so 

time-axis display means for displaying an axis 
corresponding to the time for the display of said 
video information; and 

display control means for displaying, relative to ss 
said axis, symbols corresponding to the timing 
for displaying said one or more scenes. 



6. An information processing apparatus as tn claim 1 , 
further comprising deactivation means for deacti- 
vating the synchronized display, with said video in- 
formation, of a scene of said multimedia informa- 
tion. 

7. An information processing apparatus as in claim 1 , 
further comprising mid -processing means tor set- 
ting a flag to prevent the display of a scene of said 
multimedia infonnation whose playback is synchro- 
nized to begin, with respect to said video informa- 
tion, at a point in said video infomiation which oc- 
curs before the point at which said viewer begins 
playing back said video information. 

8. An information processing apparatus as in claim 1 , 
further comprising management means for organ- 
izing and managing said multimedia infonmation in 
a hierarchy of contents, scenarios and scenes. 

9. An information processing apparatus as in claim 1 , 
further comprising transmission rate setting means 
for setting the rate at which a plurality of said scenes 
are transmitted to said viewer. 

10. An information processing apparatus as in claim 1 , 
further comprising display disabling means for set- 
ting a flag to disable said viewer's ability to prevent 
the display of said multimedia informaton. 

11. An information process ing method for creating mul- 
timedia information to accompany video informa- 
tion transmitted for display to a viewer, sad infor- 
mation processing method comprising: 

creating said multimedia information; 
identifying vtdco information with which said 
multimedia information is to be synchronized; 
and 

setting the timing for displaying one or more 
scenes of said multimedia information so as to 
cause said display to be synchronized with the 
displaying of said video information. 

12. An information processing method as in claim 11, 
wherein said multimedia information conforms to 
the MHEG standard 

13. An information processing method as in claim 11. 
wherein said multimedia information conforms to 
the ATVEF standard, 

14. An information processing method as in claim 11, 
wherein said setting comprises setting said timing 
with respect to the relative time for the display of 
said video information. 

15. An information processing method as in claim 11, 
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further comprising: 

displaying an axis corresponding to the time for 
the display ot said video infornnation; and 
displaying, relative to said axis, symtxils cor re- 5 
sponding to the timing for displaying said one 
or more scenes. 

16. An Information processing method as in claim 11, 
further comprising setting a flag for said multimedia io 
information for deactivating the synchronized dis- 
play, with said video information, of a scene of said 
multimedia information. 

17. An information processing method as in claim 11, 
further comprising setting a flag to prevent the dis- 
play of a scene of said multimedia information 
Whose playback is synchronized to begin, with re- 
spect to said video information, at a point in said 
video inforn^tion which occurs before the point at ^ 
which said viewer begins playing back said video 
information. 

18. An information processing apparatus as in claim 1 , 
further comprising organizing and managing said, 
multimedia information in a hierarchy of contents, 
scenarios, and scenes. 

19. An information processing apparatus as in claim 11 , 
further comprising setting the rate at which a plural- ^ 
ity of said scenes are transmitted to said viewer 

20. An information processing apparatus as in claim 11 , 
further comprising display disabling means for set- 
ting a flag to disable said viewer's ability to prevent 3S 
the display of said multimedia information. 

21 . A computer-readable medium having computer-ex- 
ecutable software code stored thereon for creating 
multimedia information to accompany video infor- 
mation transmitted for display to a viewer, said soft- 
ware code comprising: 

creating code for creating said multimedia in- 
formation; 45 
identifying code for identifying video informa- 
tion with which said multimedia information is 
to be synchronized ; and 
setting code lor setting the timing for displaying 
one or more scenes of said multimedia infor- so 
mation so as to cause said displaying to be syn- 
chronized with the display of said video infor- 
mation. 

22. A computer-readable medium as in claim 21, 5S 
wherein said multimedia information conforms to 
the MHEG standard. 



23. A computer-readable medium as in claim 21, 
wherein said multimedia information conforms to 
the ATVEF standard. 

24. A computer-readable medium as in claim 21, 
wherein said setting code; comprises code for set- 
ting said timing with respect to the relative lirne for 
the display of said video information. 

25. A computer-readable medium as in claim 21. 
wherein said software code further comprises: 

time-axis display code for displaying an axis 
corresponding to the time for the display of saki 
video information; and 

display control code for displaying, relative to 
said axis, symbols corresponding to the timing 
for displaying said one or more scenes. 

26. A computer-readable medium as in claim 21, 
wherein said software code further comprises de- 
activation code for deactivating the synchronized 
display, with said video information, of a scene of 
said multimedia information. 

27. A computer-readable medium as in claim 21, 
wherein sakj software code further comprises mid- 
processing setting code for setting a flag to prevent 
the display of a scene of said multimedia informa- 
tion whose playback is synchronized to begin, with 
respect to said video information, at a point in saki 
video information which occurs before the point at 
which said viewer begins playing back said video 
infornnation. 

28. A computGr-rcadabtc medium as in claim 21, 
wherein said software code turthor comprises man- 
agement code for organizing and nrranaging said 
multimedia information in a hierarchy of contents, 
scenarios and scenes. 

29. A computer-readable medium as in claim 21, 
wherein said software code further comprises 
transmission rate setting code for setting the rale at 
which a plurality of said scenes are transmitted to 
said viewer, 

30. A computer-readable medium as in claim 21, 
wherein said software code further comprise dis- 
play disabling code for setting a flag to disable said 
viewer's ability to prevent the display of said multi- 
media information. 

31. An apparatus for synchronizing the display of a vid- 
eo with multimedia infonmation corresponding to 
said video, said apparatus comprising: 

means tor specifying the duration for displaying 
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said video; 

means for specifying the period for displayinq 
each scene of a plurality of scenes of said mul- 
tinnedia information with respect to said dura- 
lion; 

means for displaying on said computer screen 
a plurality of symbols, each of sa id symbols cor- 
responding to one of said scenes and repre- 
senting the relative time and position, with re- 
spect to said duration, of the period corre- 
sponding to said scene. 

32. An apparatus as in claim 31, further comprising 
means for displaying on said computer screen, con- 
currently with said symbols, a second symbol rep- 
resenting the duration for displaying said video. 

33. An apparatus as in claim 31 , wherein said multime- 
dia information conforms to the MHEG standard. 

34. An apparatus as in claim 31 , wherein said multime- 
dia information conforms to the ATVEF standard. 

35. An apparatus for synchronizing the broadcasting of 
a video with multimedia information broadcasted 
with said video, said apparatus comprising: 

means for specifying the duration for broad- 
casting said video; 

means for specifying the period for broadcast- 
ing each scene of a plurality of scenes of said 
multimedia information with respect to said du- 
ration; 

means for displaying on said computer screen 
a plurality of symbols, each of said symbols cor- 
responding to one of said scenes and repre- 
senting the relative time and position, with re- 
spect to said duration, of the period corre- 
sponding to said scene. 

36. An apparatus as in claim 35. further comprising 
means for displaying on said computer screen, con- 
currently with said symbols, a second symbol rep- 
resenting the broadcasting duration of said video. 

37. An apparatus as in claim 34, wherein said multime- 
dia information conforms to the MHEG standard. 

38. An apparatus as in claim 34. wherein said multime- 
dia information conforms to the ATVEF standard. 

39. A computer-readable medium having computer-ex- 
ecutable software code stored thereon for synchro- 
nizing the display of a video with multimedia infor- 
mation corresponding to said video, said software 
code comprising: 

code for specifying the duration tor displaying 



said video; 

code for specifying the period for displaying 
each scene of a plurality of scenes of said mul- 
timedia information with respect to said dura- 

5 lion; 

code for displaying on said computer screen a 
plurality of symbols, each of said symbols cor- 
responding to one of said scenes and repre- 
senting the relative time and position, with re- 

'0 spect to said duration, of the period corre- 

sponding to said scene. 

40. A computer-readable medium as in claim 39, 
wherein said software code further comprises code 

'5 for displayin g on said computer screen, concurrent- 
ly with said symbols, a second symbol representing 
the duratbn tor displaying said video. 

41. A computer- readable medium as in claim 39, 
^ wherein said multimedia information conforms to 

the MHEG standard. 

42. A computer-readable medium as in claim 39, 
wherein said multimedia information conforms to 

2S the ATVEF standard. 

43. A computer-readable medium having computer-ex- 
ecutable sofhware code stored thereon for synchro- 
nizing the broadcasting of a video with multimedia 

^ information broadcasted with said video, said soft- 
ware code comprising: 

code for specifying the duration for broadcast- 
ing said video; 

^ code for specifying the period for broadcasting 

each scene ol a plurality of scenes of said mul- 
timedia inlormalion with respect to said dura- 
tion; 

code tor displaying on said computer screen a 
plurality of symbols, each of said symbols cor- 
responding to one of said scenes and repre- 
senting the relative time and position, with re- 
. spect to said duration, of the period corre- 
sponding to said scene. 

45 

44. A computer-readable medium as in claim 43, 
wherein said software code further comprises code 
lor displaying on said computer screen, concurrent- 
ly with said symbols, a second symbol representing 
the broadcasting duration of said video. 

45. A computer-readable medium as in claim 43, 
wherein said multimedia information conforms to 
the MHEG standard. 

55 

46. A computer-readable medium as in claim 43, 
wherein said multimedia information conforms to 
the ATVEF standard. 
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